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Adaptive Closed Group Caricaturing 

Technical Field 

The present invention relates to a method and system for generating caricatured 
5 images of subjects based on a closed-group of subjects. In particular the invention is 
. concerned with adaptively generating caricatures in dependence on subjects joining and 

* 

leaving the closed group. 

Background to the Invention and Prior Art 
10 Automatic caricaturing methods and systems are already known in the art. 

Brennan, S. E. in "Caricature Generator The Dynamic Exaggeration of "Faces by 
Computer." Leonardo, Vol.18 no.3, pp. 170-178 describes a computational model of 
caricature which allowed a two dimensional line drawn caricature to be generated from 
' photographs. The user traces over the original image (by placing a set of markers over the 
15 image) to generate a line drawing of the subject. An example of such an original image 
and the resulting line drawing are shown in Figure 15. Here, an original image as shown 
in Figure 15(a) results in a line drawing as shown in Figure 15(b)i 

Having obtained the line drawing of the subject, this drawing is then compared 
with a corresponding line drawing" of a -"mean" or "prototype" face, by which is meant an 
20 average face of a group usually comprising the same race, gender, and colour as the 
subject. Thus, for the white Caucasian male shown in Figure 15(a), usually a prototype 
face of an "average" white Caucasian male would be used. In some circumstances 
prototype faces from different ethnic groups may be used. 

Rowland et al in "Transforming Facial Images in 2 and 3-D". Imagina 97 - 
25 Conferences - ACTES/Proceedings, Feb, Monte Carlo, (1997), pp159-175 describe how 
a prototype face may be derived as follows. A prototype can be defined as being a 
representation of the consistencies across a collection of faces. For example, a 
prototypical male Caucasian face would contain all that is consistent about Caucasian 
faces and can be generated by calculating a mean face from a set of Caucasian faces. 
30 To derive the prototypical shape for a group of faces, the delineation data for 

each face are first "normalised"; making the faces nominally of the same size' and 
orientation. The left and right pupil centres provide convenient landmark points for this 
process. The first step is to calculate the average left and right eye positions for the whole 
population. The next step is to apply a uniform translation, scaling, and rotation to the (x, 
35 y) positions of all the feature points, thus normalising each face to map the left eye to the 
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average left eye position and the right eye to the average right eye position. This process 
maintains ail the spatial relationships between the features within each face but 
standardises face size and alignment. It. is then possible to calculate the average positions 
of each remaining template feature point (after alignment), the resulting data constituting 
5 the mean shape for the given population. A line drawing of the resulting "mean" or 
prototype face can then be obtained. An example line drawing of a mean face is shown in 
Figure 16. 

Once a prototype has been formed for a collection of faces it is possible to 
generate caricatures by accentuating the difference between an individual face and a 

10 relevant prototype. After normalising the feature location data from the prototype to the 
eye positions of an example, face, all feature points on the example face can be shifted 
away from their counterparts on the prototypical face by a given percentage, as shown in 
Figure 9. This percentage is the amount of caricature and can be thought of as 
extrapolating a morph between a prototype and the example face. If the percentage is 

15 100% then the product of the manipulation will be the prototype, if the percentage is 50% 
^ then the result will be halfway along the morph between the prototype and the example 
face, if the percentage is 0% then the example face is returned, if it is -50% then a 
caricature of the original face is the result. More generally, any percentage less than 0% 
will result in a caricatured face. Equation 1.1 expresses Brennan's caricaturing algorithm 

20 in a mathematical form. 

Q=p+b(P-S), (Equation 1.1) 

Where: 

25 Q is the feature point of the caricature model, 

P is the feature point of the individual head model, 

S is the feature point of the average mean head model and 

b is a coefficient for caricaturing. 

30 An example caricature image of the line drawing of Figure 15(b) generated 

according to the Brennan algorithm, and using a caricature coefficient b of 20% is shown 
in Figure 17. 

Benson and Perrett in "Synthesising continuous-tone caricatures." Image and 
Vision Computing, 9, 123-9 extended the technique to produce photographic caricatures 
35 using computer-morphing techniques. However the basic underlying caricaturing 



WO 2005/069226 PCT/GB 2004/0054 19 

3 

algorithm is the same as Brennan. Moreover the same caricaturing technique is easily 
extended into 3D by applying the caricaturing algorithm to every vertex or to 3D-mesh 
control points, as described in Fujiwara, T,, Nishihara, T., Tominaga, M. t Kato, K. (1998) 
"On the Detection of Feature Points of 3D Facial Image and Its Application to 3D Facial ' 

5 Caricature." and Shadbolt, A. (2003) "From 2D photographs to 3D caricatures." 
http://www.dcs.shef.ac.uk/-uQas2 respectively. When the caricaturing algorithm is applied 
to the control points of the mesh the resulting caricatured control points are then 
interpolated over the rest of the mesh to produce the 3D caricature. Figure 10 shows a 
3D-caricature (right) produced from the original 3D head model (left) and the mean model. 

10 With respect to other aspects of computational caricatures, it has also been 

shown in Rhodes, G. & Brennan, S. E. (1987), Identification and Rating of Caricatures: 
Implications for Mental Representations of Faces. Cognitive Psychology, 19, 473-497 that 
caricaturing of faces results in greater recognition of the caricature face as the subject 
than an un-caricatured face. An example of this is given in In the Eye of the Beholder - 

15 The Science of Face Perception, Bruce, V. and Young, A. 2000, ISBN 0-190852439-0 at 
p. 121-123, where photographs of two identical twins are manipulated to provide an 
"average" image of the two photographs, and differences identified between the "average" 
image and the actual photographs. Further image manipulation is then performed in 
dependence on the identified differences to exaggerate the differences, thus highlighting 

20 the differences which are available for their friends and families to learn so as to be able 

* 

to identify each twin. In effect, within this work a closed group of subjects (the two twins) is 
formed, a mean of the closed group is taken, and then the images of the subjects are 
caricatured away from this mean so as to render the images more recognisable. The 

* 

- . precise image manipulations used to perform the caricaturing are described in Chapter 5 
25 of the book. 

Although the above work by Bruce and Young introduces the concept of the 
formation of a mean image from a closed group, and the exaggeration of images of 
members of the group away from that mean so as to render them recognisable, problems 
remain in practical implementations of such techniques where the members of the group 
30 may change over time. 

Summary of the Invention 

The present invention presents a practical application of the closed group image 
exaggeration concept presented by Bruce and Young by providing a method and system 
35 for generating caricatured images of subjects who are members of a closed group, where 
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the membership of the closed group may change over time. More particularly, the present 
invention presents a method and system for generating caricatured images of subjects 
wherein the caricatured image of each subject is adapted when subjects join and leave a 
closed group of which they are members. By adapting the caricatured images in this 
5 manner, it can be ensured that each caricatured image remains maximally distinguishable 
' from the other images in the group, thus ensuring that recognition of the images may be 
maintained at a high level. 

In view of the above, from a first aspect there is provided a method of generating 
a caricatured image, comprising the steps of: 
10 storing image representations of subjects and corresponding respective 

caricatured image representations of the subjects; 

receiving an image representation of a new subject; and 

generating a caricatured image representation of the new subject in dependence 
on the stored image representations of the subjects and the received image 
1 5 representation of the new subject. 

In the first aspect, when a new member wishes to join a closed group of subjects, 
a caricatured image of the member exaggerating the differences between the new 
member and the existing members can be created by taking into account the images of 
the existing members when generating the caricature of the new member. In this way the 
20 caricature image of the new member will be maximally distinguishable from the other 
caricature image representations. 

From a second aspect, the invention further provides a method of generating 
caricatured images, comprising the steps of: 

storing image representations of subjects and corresponding respective 
25 caricatured image representations of the subjects; 

' receiving-an image representation of a new subject; and 

generating replacement caricatured image representations of the subjects in 
dependence oh the stored image representations thereof and the received image 
representation of the new subject. 
30 In the second aspect, when a new member wishes to join a closed group of 

subjects, replacement caricatured images of the existing members exaggerating the 
differences between the members can be created by taking into account the images of the 
existing members and the new member. In this way the replacement caricature images of 
the members will be maintained as maximally distinguishable from each other. 
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Within the second aspect there preferably further is included the step of 
generating a caricatured image representation of the new subject in dependence on the 
stored image representations of the subjects and the received image representation of the 
new subject. In such a case both replacement caricature images of the existing member, 
5 and a new caricatured image of the new member are created, all of which are maximally 
distinguishable from each other. 

In either the first or second aspects of the invention the generating steps 
preferably further comprise generating the caricatured image represehtation/s in weighted 
dependence on the received image representation of the new subject, wherein a 
10 weighting factor associated with the new image representation generally increases with 
time, and the caricature image representations are re-generated each time the weighting 
factor is adapted. This will avoid dramatic and potentially disturbing step changes to any 
particular member's caricature image representation, allowing the impact of the new 
member to be gradually incorporated into the group over time. 
15 From a third aspect there is further provided a method of generating caricatured 

images, comprising the steps of: 

storing image representations of subjects and corresponding respective 
caricatured image representations of the subjects; 

receiving a leave signal indicating a particular one or more of the subjects for 
20 which image representations are stored; and 

generating replacement caricatured image representations in dependence on the 
stored image representations of the subjects at least partially discounting the image 
representations of the indicated subjects. 

Within the third aspect where caricature images of a group of subjects are 
25 already in existence, when a subject wishes to leave the group replacement caricature 
images can be generated' based on the remaining group members, but at least partially 
taking into account the image representations of the subjects who are to leave the group, 
if desired. This allows for the caricature image representations to be modified such that 
they may remain maximally distinguishable from each other, but by at least partially 
30 including the image representations of the members who have left the group in the 
generation process step changes in appearance may be avoided, if desired. 

Preferably, within the third aspect the image representations of the indicated 
subjects are generally increasingly discounted over time in accordance with a weighting 
factor, wherein the caricature image representations are re-generated each time the 
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weighting factor is adapted. This allows for the subjects who have left the group to have a 

decreasing impact on the generated caricatures over time. 

From a fourth aspect the present invention further provides a computer program 

or suite of computer programs arranged such that when executed by a computer system 
5 it/they cause the computer system to operate according to any of the aforementioned first, 

second, or third aspects. 

Moreover, from a fifth aspect, there is also provided a computer readable storage 

medium arranged to store a computer program according to the fourth aspect of the 

invention. The computer readable storage medium may be any magnetic, optical, 
10 magneto-optical, solid-state, or other storage medium capable of being read by a 

computer. 

From a sixth aspect the present invention further provides a system for 
generating a caricatured image, comprising: 

storage means arranged in use to store image representations of subjects and 
1 5 corresponding respective caricatured image representations of the subjects; 

input means for receiving an image representation of a new subject; and 
processing means arranged in use to generate a caricatured image 
representation of the new subject in dependence on the stored image representations of 
the subjects and the received image representation of the new subject. 
20 Within the sixth aspect corresponding advantages and further features as 

previously described in respect of the first aspect are obtained. 

From a seventh aspect the invention also provides a system for generating 
caricatured images, comprising: 

storage means arranged in use to store image representations of subjects and 
25 corresponding respective caricatured image representations of the subjects; 

input means for-receiving an image representation of a new subject; and 
processing means arranged in use to generate replacement caricatured image 
representations of the subjects in dependence on the stored image representations 
thereof and the received image representation of the new subject. 
30 Within the seventh aspect corresponding advantages and' further features as 

previously described in respect of the second aspect are obtained. 

Additionally, from an eighth aspect of the invention there is provided a system for 
generating caricatured images, comprising: 

storage means arranged in use to store image representations of subjects and 
35 corresponding respective caricatured image representations of the subjects; 
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means for receiving a leave signal indicating a particular one or more of the 
subjects for which image representations are stored; and 

processing means arranged in use to generate replacement caricatured image 
representations in dependence on the stored image representations of the subjects but at 
5 least partially discounting the image representations of the indicated subjects. 

Within the eighth aspect corresponding advantages and further features as 
previously described in respect of the third aspect are obtained. 

Brief Description of the Drawings 
10 Further features and advantages will become apparent from the following 

description of embodiments of the present invention, presented by way of example only, 

and by reference .to the accompanying drawings, wherein like reference numerals refer to 

like parts, and wherein: 

Figure 1 is an illustration of a computer system which may form the operating 
15 environment for the present invention, together with an indication of the various computer 

programs and data which are required by the embodiment of the invention; 

Figure 2 is a flow diagram illustrating steps performed by the described 

embodiments of the invention; 

Figure 3 is a flow diagram illustrating steps performed by the described 
20 embodiments of the invention; . - 

Figure 4 is an example caricature image representation generated by the 
described embodiment of the invention; 

Figure 5 is a second example caricature image representation generated by the 
described embodiment of the invention; 
25 Figure 6 is a third example of a caricature image representation generated by 

the described embodiment of the invention; 

Figure 7 is a screen shot of an example application program which may make 
use of the embodiment of the invention; 

Figure 8 is further screen shot of an example application program which may 
30 make use of the embodiment of the invention; 

Figure 9 is a diagram illustrating the concept of the Brennan characturing 
algorithm of the prior art; 

Figure 10 is an example of caricature image representations generated using 
prior art techniques; 
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- Figure 11 is a flow diagram illustrating steps performed by the described 
embodiments of the invention; 

Figure 12 is a graph of a time dependent weighting function which may be used 
by preferred embodiment of the invention; 
5 Figure 13 is a graph of a time dependent weighting function which may be used 

by preferred embodiment of the invention; 

Figure 14 is a flow diagram illustrating steps performed by the described 
embodiments of the invention; ■ 

Figure 1 5(a) is a photographic image representation of a Caucasian male; 
10 Figure 15(b) is a drawing of. an example input image representation 

corresponding to Figure 15(a), and of the type used by the prior art Brennan algorithm, 
and also by some embodiments of the invention; 

Figure 16 is an example mean face of the,prior art; 

Figure 17 is an example caricatured image representation generated using the 
15 input image of Figure (b), the mean face of Figure 16, and the prior art Brennan algorithm; 

Figure 18 is a table of caricatured image representations illustrating the effects of 
using a mean face generated by a closed group of members; and 

Figure 1 9 is a table of caricatured image representations generated according to 
the embodiments of the invention, and which illustrates the changes in caricatured image 
20 representations as members join and leave the closed group. 

Description of an Embodiment 

An example embodiment of the invention will now be described with respect to 
Figures 1 to 3, and 1 1 to 14. Example output caricatured image representation which may 

25 be generated by the described embodiment of the invention will further be described with 
respect to Figures 4 to.Bf and then an example application which may make use of the 
concept of the invention will be described with respect to Figure 7 and 8. 

The embodiment of the invention provides an adaptive closed group caricaturing 
method and system which is primarily implemented in software, and which employs 

30 known prior art computational caricaturing algorithms, such as those described in Brennan 
referenced earlier, or the applicant's own "feature based caricaturing" algorithm described 
in the present applicant's earlier co-pending United Kingdom patent application No. • 
0311208.3 filed 15 th May 2003, a copy of which is filed concurrently herewith so as to be 
available on the file of this application. In particular, the embodiments of the invention 
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provide for the caricature image representations of members of the closed group to be 
adapted in dependence on members joining and leaving the group. 

In order to provide the above functionality, according to an embodiment of the 
invention, and with reference to Figure 1, a general purpose computer system 10 is 
5 provided, which is equipped with a computer readable storage medium such as a hard 
disk drive, optical drive, dvd drive, or the like, 12, upon which is stored various data and 
computer executable programs which in use control the general purpose computer system 
to perform the invention. More particularly, provided on the computer readable storage 
medium 12 is a caricature generator program 24, a mean face generator program 22, a 

10 control program 20, a subject image store 14, a caricature image store 16 and a data 
store 18. The operations of each of the programs 20, 22, and 24, and the contents of the 
various stores 14, 16, and 18 will be described in detail later. 

As mentioned above, the aim of the embodiments of the invention are to provide 
an adaptive closed group caricaturing process and system, and Figure 14 illustrates the 

15 general steps involved in such a method and system. In particular, with reference to 
Figure 14, at step 14.2 the first operation performed by embodiments of the invention is to 
form a closed group of members, and how this is performed will ultimately depend upon 
the application to which the embodiments of the invention are put. For example, in a 
group conferencing application or online gaming application, individual users may register 

20 with a central server or the like, and transmit facial images of themselves to the server for 
processing into caricatures. However this step is performed, at the end of step 14.2 a 
closed group of members, and their facial image representations has been obtained. 

The next step to be generally performed at step 1 4.4 is to calculate caricature 
image representations for the group members. The individual steps involved in step 14.4 

25 are illustrated in Figure 11. Here, at step 11.2 the images of the members which form a 
closed group are-receiVed. Next, at step 11.4 these images are used to calculate a new 
mean or "prototype" facial image, and this may be performed in the same manner as 
described previously with respect to the Brennan prior art. The output of step 11.4 is 
therefore a mean facial image which corresponds to the mean of the closed group of 

30 images received at step 11.2. Therefore the mean face calculated at step 11.4 is 
influenced only by the facial image representations of the closed group members, and by 
no other facial image representations. It is the average facial image of the closed group 
members but of no one else. 

Having calculated such a closed group mean face, at step 11.6 a new caricature 

35 image is generated for each member of the closed group using the mean face calculated 
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at step 11.4. As mentioned previously, the precise computational caricature algorithm 
which may be employed in embodiments of the invention may be any of those already 
known in the art, such as the Brennan algorithm, or the applicant's own "feature based 
caricaturing" algorithm, referenced earlier. Commonly, such algorithms allow the user to 
5 set a caricaturing coefficient (b) which determines the level of caricaturing applied, and 
where this is necessary or available step 11.6 preferably includes steps where the user is 
asked to set such a level. Howsoever the caricaturing fs performed, the results of step 
11.6 are, however, that a caricatured image representation is obtained for each member 
of the closed group using the mean face of the closed group. As discussed in the 
10 introduction, the use of the closed group mean face ensures that the resulting caricatured 
image representations are maximally distinguishable from each other. 

Once each new caricature has been generated, the caricatured images can be 
output for display and/or storage at step 1 1.8. 

Returning to. Figure 14, once the process steps of Figure 11 have been 
15 performed at step 14.4, the result is that a closed group of members has been formed, 
and that caricature images are available for each of the members in the closed group. 
According to embodiments of the invention, however, once the initial closed group 
caricature image representations have been obtained, at step 14.6 the obtained caricature 
image representations are adapted whenever members join or leave the closed group. As 
20 discussed previously, this ensures that the caricatured image representations remain 
maximally distinguished from each other, and hence the recognisability of each image is 
maintained even though members may have joined or left the closed group. 

In view of the above-described overall operation of embodiments of the invention, 
returning to Figure l.the above described operation can be related to the various 
25 programs and stores provided on the computer readable storage medium 12. More 
particularly, in order to form the initial closed group of caricature image representations, 
the facial images of the members of the closed group are stored in the subject image 
store 14. The mean face generator program 22 then operates at step 11.4 to calculate 
the new mean face using the stored subject images, and the new closed group mean face 
30 is stored in the data store 18. The closed group mean face is then used at step 11.6 
together with each respective closed group member facial image retrieved from the 
subject image store 14 by the caricature generator program 24 to generate each new 
caricature image representation. Each generated caricature image representation is 
stored in the caricature image store 16. The operation of each of the caricature generator 
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program 24 and the mean face generator program 22 is co-ordinated overall by the 
control program 20. 

Example closed group caricature image representations are shown in Figures 4, 
5, and 6. In Figure 4, the closed group comprises the two members whose facial images 
5 are highlighted on the bottom row of the images in the window. The resulting virtual 
reality style U 3D D caricature image representation is then shown directly above. Similarly, 
in Figures 5, and Figure 6, the closed group which has been used as the basis for the 
mean face used in the caricaturing process are illustrated by the highlighted facial images 
at the bottom of the each window. From a comparison of each of the caricatured image 

10 representations of Figures 4, 5, and 6, it will be seen that for each different closed group 
which is used to form the mean face, a different caricatured image representation of the 
same person is achieved. Although the differences are slight, as demonstrated by the 
work of Bruce and Young concerning the identical twins even the slight exaggeration of 
differences in appearances between group members will aid in the recognition thereof. 

15 A further example application of the use of closed group caricaturing is shown in 

Figure 7. In particular, Figure 7 illustrates the graphical user interface to a group 
messaging application, wherein each member of the group is represented by a facial 
image as shown. Figure 7 illustrates the facial images when not subject to caricaturing. 
In Figure 8, however, the basic facial images of Figure 7 have been replaced by 

20 caricatured image representations for each member of the closed group, which have been 
generated using a mean face obtained by taking the mean of the facial images of Figure 
7. Comparing Figure 8 to Figure 7, it will be seen that the distinctive features of each of 
the facial images of the members of the group have been emphasised in Figure 8, thus 
rendering each individual member more recognisable. As an aside, the group messaging 

25 application operates by displaying a message to the user, together with an indication of 
which of the group_ members sent the message. A user can send his own message by 
filling in the input box, and clicking on the send button. Such a simple messaging 
application is well known in the art, but the graphical. user interface using the closed group 
caricatured image representations improves the recognisability of each of the individual 

30 members. 

Thus far, we have concerned ourselves with how the initial group caricature 
image representations are obtained, and how they may be used. Embodiments of the 
invention, however, are concerned with how the closed group caricature image 
representations may be adapted, when members of the closed group join or leave the 
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group. How member joins or leaves are handled will now be described with respect to 
Figures 2 and 3 respectively. 

With reference to Figure 2, assume that a closed group of members has been 
formed using a particular application (such as the messaging application discussed 
.5 above), and that caricatured image representations are being used to represent each of 
the members in that application. Moreover, assume also that a server is being used to 
host the application, and is capable of computing new caricatured image representations 
when required. 

With reference to Figure 2, assume that a new member is to join the closed 
10 group. In this case, within this embodiment the new member signals to the server which 
is hosting the application which the closed group is using, sending an indication of their 
wish to join the group, and a facial image of the new subject at step 2.2. The server 
hosting the application receives this join signal and at step 2.4 receives the image of the 
subject. 

15 Having received the image of the new subject, in order to maintain the 

caricatured image representations of the closed group members as being maximally 
distinguished from each other, it is necessary to recalculate the mean face which is used 
to generate the caricatured image representations to take into account the image of the 
new subject received at step 2.4. If, however, the received new facial image is allowed to 

20 have the same impact on the new mean face to be calculated as every other facial image 
of the present group members, then it may be that the caricatured image representations 
of the present group members would undergo a step change when the new mean face is 
used to generate replacement caricatured image representations. Whilst in some 
embodiments of the invention such a step change may be acceptable, in the preferred 

25 embodiment a time dependent weighting function is employed which weights the 
contribution of the newfacial image to the mean as a generally increasing function over 
time. An example of such a function is shown in Figure 12, which is of a general raised 
co-sine form, but it should be noted that embodiments of the invention are not limited to 
such a form of function, and that any time dependent generally increasing function may be 

30 used. 

To apply the function to. the calculation of the new mean face, at step 2.6 the 
mean face generator program 22 accesses the data store 18, which may store a look up 
table or the like which embodies the function of Figure 12. In the case of a look up table, 
weighting values to be applied to the received new facial image are indexed against time 
35 since the join request was received, such that the first time the^new mean face is 
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calculated taking into account the new received facial image, the weighting of the facial 
image on the resulting new mean face is small. That is, at step 2.6 the mean face 
generator program 22 calculates a new mean face using the facial images of the existing 
closed group members, but adding only a small contribution to the mean from the new 

5 facial image of the new member who is joining the closed group. 

Having obtained such a new mean face, at step 2.8 the caricature generator 
program 24 generates a new caricature image for the new closed group member, and 
stores the newly generated caricature image representation in the caricature image store 
16. As mentioned previously, the caricature generator program 24 runs any known 
10 caricaturing algorithm, such as the Brennan algorithm, for example, to produce the 
caricatured image representations. Where the algorithm requires or allows a caricature 
coefficient value to be set by a user, then a user interface to permit such operation is 
provided by the caricature generator program 24. 

Next, at step 2.10 the caricature generator program 24 generates replacement 

15 caricature images for each of the existing closed group members, using the new mean 
face just calculated at step 2.6. The replacement caricature image representations are 
also stored in the caricature image store 16. Once a replacement caricature image 
representation has been generated for each of the existing closed group members, all of 
. the newly generated caricature image representations stored in the caricature image store 

20 may be output for display and/or storage or other use by the application using the 
embodiment of the invention. 

At this point, therefore, replacement caricature image representations have been 
generated for each of the existing members, and a new caricature image representation 
has been generated for the new member who is joining the closed group, but in each case 

25 the impact of the facial image of the new member on the mean face which has been used 
to generate the .new caricature image representation images is minimal. 

Following step 2.12, processing proceeds to step 2.14 wherein an evaluation is 
performed to determine whether or not the facial image of the new closed group member 
has yet been given the maximum weighting in the mean face calculation. If this is the 

30 case, i.e. the new member's facial image is given an equal weighting in the mean face 
calculation to each of the existing member's facial images, then within the context of the 

* 

embodiment the new member has become fully integrated into the closed group, and 
hence processing may end. If this is not the case, however, and the new member's facial 
image has not been given an equal weighting to the existing member's facial images in 
35 the mean face calculation, then processing proceeds to step 2.16, wherein the weighting 
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given to the new member's facial image is increased. In this case, where the weighting 
function is represented by a look up table, the next value may be accessed and used in 
the subsequent mean face calculation. However the weighting function is implemented, 
however, it should be understood that on each iteration of the loop comprising steps 2.6 to 
5 2.16, the weighting given to the new member's facial image in the mean face calculation 
should generally increase. 

Once the weighting value has been incremented at step 2.16, processing 
proceeds back to the mean face calculation at step 2.6, wherein a new mean face is 
calculated, and the increased weighting of the new member's facialimage is used in the 
10 new mean face calculation. Processing then proceeds through the steps 2.8, 2.10, and 
2.12 back to the evaluation of step 2.14, as previously described. 

From the above description and Figure 2 it should thereforebe apparent that the 
loop formed by steps 2.6 to 2.16 is repeated until the new member has been fully 
integrated into the closed group. By allowing the impact of the new member's facial 
15 image on the mean face calculation to steadily increase over time, and by generating new 
replacement caricature images for the existing members and the new member each time 
the weighting is changed, drastic step changes in the appearances of the existing 
members may be avoided, and the appearances will instead steadily adapt to the 
inclusion of the new member. In this way the advantages of using a closed group mean to 
20 maintain maximum recognisabiiity of each of the caricatured images is maintained, but 
disturbing large step changes in the appearances of the existing caricatured image 
representations are avoided. 

Similar considerations may also be applied when existing members leave the 
closed group. The processing performed by embodiments of the invention when an 
25 existing member leaves the group will be described next with respect to Figures 3, and 13. 

With reference first to Figure 13, Figure .13 shows a weighting function which 
relates the contribution of the facial image of a member who is leaving the group to the 
mean face over time, in a time dependent generally decreasing manner. As with the 
joining weighting function of Figure 12 used when new members joined the closed group, 
30 the leaving weighting function of Figure 13 is a generally raised co-sine function, although 
a function of this form is not essential, and it should be understood that any time 
dependent generally decreasing function may be used. Similarly, as with the joining 
weighting function of Figure 12, the leaving weighting function of Figure 13 may be 
implemented by way of a look-up table or the like stored in the data store 18. Moreover, in 
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other embodiments of the invention where drastic step changes in the caricature image 
representations are not a concern, such a weighting function need not be used. 

Turning now to Figure 3, this illustrates the process steps which are performed 
when an existing closed group member wishes to leave the group. Firstly, at step 3.2 the 

5 member who wishes to leave the closed group signals the host running the closed group 
application of his intentions to leave. The host receives the leave signal at step 3.2, and 
this causes the host to use the mean face generator program 22 to calculate a new mean 
face at step 3.4 using the facial images of the group members who are to remain, and 
applying the weighting of Figure 13 to determine the contribution that the facial image of 
10 the member who is to leave the group is to have on the new mean face calculation. 
Therefore, at step 3.4 the mean face generator program 22 accesses the leave weighting 
function in the data store 18 to determine the weighting value which should be applied in 
dependence on the time from when the leave signal is received, and then applies the 
determined weighting value to the facial image of the member who is to leave the group in 

1 5 the new mean face calculation. The result of step 3.4 is therefore that a new mean face is 
calculated on the basis of the facial images of the members who are to remain within the 
group, and also of the facial image of the member who is to leave the group, although 
partially discounted. 

Having obtained a new mean face, at step 3.6 replacement caricature image 
20 representations are generated for the closed group members who are to remain, by the 
caricature generator program 24. The replacement caricature image representations are 
stored in the caricature image store 16, and also output for display and/or storage, or 
other use by the application using the embodiment at step 3.8. 

Following step 3.8, at step 3.10 an evaluation is performed to determine whether 
25 or not the minimum weighting (i.e. zero) is being applied to the facial image of the member 
who is to leave "the group, in the new mean face calculation of step 3.4. If the minimum 
weighting has been applied, then the result is that the facial image of the member who 
has left the group will have no impact at all on the new mean face calculation i.e. the 
mean face generated will simply be the mean of the remaining group members. In such a 
30 case the member who has left the group will then have been extricated completely from 
the group. 

If the evaluation at step 3.10 determines the minimum weighting is not being 
applied, however, at step 3.12 the weighting applied to the left member's facial image is 
decremented in accordance with the weighting function of Figure 13. Processing then 
35 proceeds back to step 3.4 wherein a new mean face is calculated, -using the new lower 



WO 2005/069226 PCT/GB2004/005419 

16 

weighting to determine the extent to which the left member's facial image impact the mean 
face. The processing loop of steps 3.4 to 3.12 is then continuously repeated until the 
weighting is reduced to zero. 

The effect of such processing is that step changes in the appearances of the 
5 remaining group members are avoided, but ultimately over time the facial image of the 
member who has left the group is no longer represented in the mean face, such that the 
mean face used to generate the replacement caricature image representations is simply 
the mean of the remaining members. Such operation ensures that even when members 
leave the closed group, the caricature image representations may be adapted so as to 
10 remain maximally recognisable from each other. 

Example output caricatured image representations of embodiments of the 
invention which illustrate the adaptive closed group caricaturing effect are shown in 
Figures 18 and 19. These caricatured image representations were generated using line 
drawing image representations of the closed group members as input images, and by 
15 using the Brennan algorithm as the caricaturing technique, with a caricaturing coefficient 
(b)of-1 (-100%). 

The images in the table of Figure 18 demonstrate the advantage of caricaturing 
against a closed-group mean, rather than a standard mean. The input images are shown 
on the top row, the second and third rows show the caricatures with respect to the left- 

20 most mean. In the second row the mean is that supplied with the FaceBend application 
(G. M. Townsend, 1997 - http://www. cs.arizona.edu/icon/nbrarv/qproas/facebend.htm^ and is 
comparable to mean faces used in the literature with the Brennan algorithm. In the second 
row the mean has been calculated from the input images of the faces in this group. This 
closed-group mean captures the common elements of this set; these examples are of 

25 males, between the ages of approximately 18-32 years. As such the shorter hair, thinner 
eye-brows, larger -noses; higher hair lines are captured in the mean and are not 
emphasised in the resultant caricatures. 

The table of images in Figure 19 demonstrates the dynamic behaviour as faces 
join and leave the group. In the left-most column the current mean face is presented, 

30 calculated as a mean of the closed group consisting of the members whose caricatured 
image representations are changed with the membership of the group can be observed. 

The applications of the embodiments of the invention are many. For example, 
the invention may find use in messaging applications such as that described previously, or 
in other visual conferencing applications. Additionally, it is envisaged that the invention 

35 may also find much gse in online gaming applications, wherein individual players play 
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either in teams or separately, but where individual players may join or leave the game at 
any time. Where a player is represented in the game by an avatar, the facial features of 
the avatar may be caricatured in accordance with the principles of the invention such that 
each player's avatar remains maximally distinguishable from every other. 

5 Further examples of possible uses are in address book applications. Here, an 

image of a person may be used to represent that person's details in an address book 
application. In order to ensure that the images- are maximally recognisable, an 
embodiment of the present invention may be used to, provide caricatured image 
representations for use by the application, taking the entries in the address book as the 

10 closed group. When an entry is added to the address book then this is the same as a new 
member joining the closed group of members consisting of the existing address book 
entries, and the embodiment of the invention is used to compute the new caricatured 
image representation of the new entry, and also to re-compute the existing caricatured 
image representations, as described above. Similarly, when an entry is to be deleted from 

15 the address book then this is the same as a member leaving the closed group of address 

book entries, and hence the member leave procedures described above may be used to 

* 

re-compute caricatured image representations of the remaining entries. 

Unless the context clearly requires otherwise, throughout the description and the 
claims, the words "comprise", "comprising" and the like are to be construed in an inclusive 

* 

20 as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, 
but not limited to". 

Moreover, for the avoidance of doubt, where reference has been given to a prior 
art document or disclosure, whose contents, whether as a whole or in part thereof, are 
necessary for the understanding of the operation or implementation of any of the 
25 embodiments of the present invention by the intended reader, being a man skilled in the 

0 

art, then said cdntents~should be taken as being incorporated herein by said reference 
thereto. 



